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(54) Motion analysis of moving images 



(57) A motion analysis system is described that determines and utilises the second differential d 2 (CS)/dx 2 of a 
correlation surface in identifying correlation maxima 16 that are sufficiently sharp (i.e. second differential exceeds 
a threshold Th 8 ) to be valid. In a surface with more than one correlation maximum 14, 16 the system scans a 
complete correlation surface and determines the best (CS b ) and next best (OS,*) correlation maxima in that 
surface. The correlation surface is rejected unless the difference between these two maxima exceeds a 
predetermined threshold TH A and the best maximum is also the point of maximum correlation included the 
correlation surface. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
The print reflects an assignment of the application under the provisions of Section 30 of the Patents Act 1977. 
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MOTION ANALYSIS OF MOVING IMAGES 

This invention relates to motion analysis of moving images. More 
particularly, but not exclusively, this invention relates to motion 
analysis of moving images as part of motion compensated moving image 
standards conversion, e.g. high definition television format to 50Hz 
2:1 PAL television format. 

A motion compensated video standards converter is described in 
British Published Patent Application GB-A-2 231 7^9 (Sony Corporation) . 
The system of GB-A-2 231 7^9 performs motion compensated interpolation 
to derive a sequence of output images from a sequence of input images 
having a different format. As illustrated in Figure 1 of GB-A-2 231 
749 (a copy of which is reproduced herein as Figure 1 of the 
accompanying drawings), the standards converter comprises a sequence of 
processing units. 

A video signal is supplied to an input terminal 1. The input 
terminal 1 is connected to a progressive scan converter 2 in which the 
input video fields are converted into video frames which are supplied 
to a direct block matcher 3 wherein correlation surfaces are created. 
These correlation surfaces are analyzed by a motion vector estimator k, 
which derives and supplies motion vectors to a motion vector reducer 5» 
wherein the number of motion vectors for each pixel is reduced, before 
they are supplied to a motion vector selector 6 which also receives an 
output from the progressive scan converter 2. Any irregularity in the 
selection of the motion vectors by the motion vector selector 6 is 
removed by a motion vector post processor 7 from which the processed 
motion vectors are supplied to and control an interpolator 8, which 
also receives an input from the progressive scan converter 2, The 
output from the interpolator 8, which is a standards-converted and 
motion-compensated video signal, is supplied to an output terminal 9- 

The quality of output of such standards converters is affected by 
the accuracy with which motion vectors are identified. Measures which 
improve the accuracy with which motion vectors are identified will 
produce an improvement in the quality of the output. 

The invention provides apparatus for motion analysis of moving 
images, said apparatus comprising: 

means for calculating a correlation surface representing image 
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correlation between a search block of image data within a current image 
and portions of a temporally adjacent image displaced by differing 
displacement vectors from said search block; 

means for identifying points in said correlation surface 
5 corresponding to correlation maxima; 

means for calculating a value of a second differential of said 
correlation surface in at least one direction at least said points of 
correlation maxima; 

means for comparing said at least one value with a predetermined 
10 second differential threshold value in a threshold test; and 

means for rejecting any points of correlation maxima that fail 
said threshold test. 

The invention recognises that sharper correlation maxima are more 
likely to relate to valid motion vectors than less sharp correlation 
15 maxima. This is exploited by including a threshold test at candidate 

points of correlation maxima to determine if the second differential of 
the correlation surface at these points exceeds a predetermined 
threshold value . The second differential gives a good measure of 
correlation maximum sharpness with an increase in magnitude of the 
20 second differential corresponding to an increase in sharpness. Points 
of correlation maxima that fail this test will be rejected as potential 
source for a motion vector. 

A second differential measures the sharpness of a correlation 
maxima along a particular line. Whilst using the second differential 
25 along a single line will produce a course test for sharpness, it may 
produce a misleading result in the case of a correlation maxima that is 
sharp in one direction but not sharp in another direction, A preferred 
compromise between ease* of implementation and the effectiveness of the 
test is one in which said means for calculating calculates a value of 
30 a second differential of said correlation surface in two directions. 

It will be appreciated that the effectiveness of the test 
provided by determination of the second differential in the two 
directions will be increased in embodiments in which said two 
directions are orthogonal. 
35 The co-ordinate systems relative to which the second 

differentials are calculated can vary. Embodiments of the invention 
utilising polar co-ordinates and calculating second differentials along 



the r and 8 axis would be possible. However, in preferred embodiments 
of the invention said two directions correspond to cartesian coordinate 
directions in said correlation surface. 

One way in which points of correlation maxima may be identified 
is to test the values surrounding a given point to see if they all 
indicate a lower correlation. If this is the case, then the point will 
be a correlation maximum. However, in preferred embodiments of the 
invention said means for identifying calculates a value of a first 
differential of said correlation surface in two directions and 
identifies as correlation maxima points at which said values both pass 
from indicating a correlation increase to indication a correlation 
decrease. 

The use of the first differential to identify points of 
correlation maxima is complementary to the use of the second 
differential to test those points since determination of the second 
differential involves previously determining the first differential. 

A feature of preferred embodiments of the invention that allows 
unsuitable correlation surfaces to be detected is provided by means for 
detecting a correlation magnitude maximum point in said correlation 
surface at which correlation is at a maximum; and means for rejecting 
said correlation surface for use in motion analysis if said correlation 
magnitude maximum point does not correspond to a point of a correlation 
maximum that passed said threshold test. If the correlation magnitude 
maximum point in a surface does not correspond to a sufficiently sharp 
correlation maximum, then it is unlikely that reliable motion vectors 
can be determined from that surface. This feature allows such 
correlation surfaces to be detected and rejects them as potential 
sources of motion vectors. 

Another problem that can arise in the analysis of correlation 
surfaces results from the occurrence of multiple, typically closely 
spaced, points of correlation maxima resulting from alias effects, A 
problem that can also arise is having a correlation maxima that is 
sharp in one direction but not sharp in another (i.e. a trough minimum 
in a non-inverted surface). In order to help in identifying such 
problem situations, preferred embodiments of the invention comprise 
means for determining, in a correlation surface with at least two 
correlation maxima, a difference value of correlation magnitude between 



a point of a correlation maximum having greatest correlation magnitude 
and a point of a correlation maximum having next greatest correlation 
magnitude; and means for rejecting said correlation surface for use in 
motion analysis if said difference value is less than a predetermined 
difference threshold value. 

This feature can be thought of as testing for the "uniqueness" 
of points of correlation maxima. If a correlation surface does not 
include a point of correlation maximum that is sufficiently distinct 
from any others (e.g. due to aliasing or a trough) then that 
correlation surface will be rejected as a source of motion vectors. In 
a surface where the two greatest correlation maxima have the same 
correlation magnitude, one will be assigned as the greatest and one 
will be assigned the next greatest. In this case the difference value 
will be zero and the surface will fail the test. 

It will be appreciated that this invention is particularly 
applicable to the field of motion compensated image standards 
conversion. Such standards conversion typically takes place prior to 
an image signal being broadcast and in the broadcast field the 
requirements for image quality are particularly strict. 

The invention also provides a method of motion analysis of moving 
images, said method comprising the steps of: 

calculating a correlation surface representing image correlation 
between a search block of image data within a current image and 
portions of a temporally adjacent image displaced by differing 
displacement vectors from said search block; 

identifying points in said correlation surface corresponding to 
correlation maxima; 

calculating a value of a second differential of said correlation 
surface in at least one direction at least said points of correlation 
maxima; 

comparing said at least one value with a predetermined threshold 
value in a threshold test; and 

rejecting any points of correlation maxima that fail said 
threshold test. 

An embodiment of the invention will now be described, by way of 
example only, with reference to the accompanying drawings in which: 

Figure 1 is a block diagram of a motion compensated moving image 



standards converter of the type disclosed in GB-A-2231749; 
Figure 2 shows an inverted correlation surface; 
Figure 3 shows a section through two aliased correlation maxima; 
Figure 4 shows a trough correlation maximum; 
Figure 5 illustrates a line through a correlation surface with 
corresponding first and second derivatives; 

Figures 6, 7 and 8 illustrate a circuit for identifying and 
testing correlation maxima; and 

Figures 9 and 10 are flow" diagrams illustrating the tests 
performed by the circuits of Figures 6, 7 and 8. 

Figure 2 illustrates an inverted correlation surface. The X and 
Y axis represent the position within a search area in one frame where 
tests for a matching pattern to that from a search block within a 
temporally adjacent frame were made. This correlation test is 
performed by the direct block matcher 3 of Figure 1 and involves the 
summing of the difference in amplitude values for the pixels within the 
search block compared to the corresponding pixels at that position in 
the search area. A high resulting value represents low correlation and 
a low resulting value represents high correlation. The correlation 
surface shown in Figure 2 has had its Z axis inverted. Accordingly, a 
maximum in Figure 2 corresponds to a correlation maximum. Without this 
inversion, a minimum in the correlation surface corresponds to a 
correlation maximum. 

The correlation surface of Figure 2 shows two closely spaced 
correlation maxima 2, 4. Such closely spaced correlation maxima are 
typically produced due to alias effects. The correlation surface of 
Figure 2 also includes a ridge 6 (in a non-inverted correlation surface 
this would be a trough) . Such a ridge may be the result of correlation 
within an image frame rather than between image frames and can have the 
effect of disrupting the motion vector identification process. 

Figure 3 illustrates a section through a non-inverted correlation 
surface. This surface includes two distinct correlation maxima 8 t 10. 
Both these correlation maxima 8, 10 have the same correlation magnitude 
Mag. In this circumstance, it may not be possible to determine which 
of the correlation maxima 8, 10 represents the true motion vector. 

Figure 4 illustrates a correlation surface including a trough 
correlation maximum 12. When such a correlation surface is searched 
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for correlation maxima, a succession of points of correlation maxima 
running along the base of the trough 12 will be identified. The system 
may not be able to determine which of these points in fact represents 
the true motion vector. 
5 Figure 5 illustrates a section through a correlation surface 

along a line x together with the first and second differentials along 
that line. The section through the correlation surface shown in graph 
a) includes two minimum points 14, 16 corresponding to points of 
correlation maxima. The correlation maximum at point 16 is sharper 

10 than that at point 14. The magnitude of the correlation value at point 
16 is sufficiently different to that at point lA such that an aliasing 
threshold value TH A is exceeded. Accordingly, the correlation surface 
would pass the test that its greatest magnitude correlation maximum was 
sufficiently unique so as not to be effected by aliasing or a trough. 

15 Graph b) shows the first differential along the line x of the 

correlation surface shown in a). At the points corresponding to the 
correlation maxima l*f, 16, the value of the first differential is zero 
and is passing from negative to positive. In contrast, at point 22 
corresponding to a correlation minimum 24, the value of the first 

20 derivative is zero but this time passing from positive to negative. It 

will be understood that if the correlation surface were inverted, then 
the sign change at a zero crossing indicating a correlation maximum 
would also be inverted. It will be seen that the relative sharpness of 
the correlation maximum 16 has resulted in a rapid change in the first 

25 differential about point 20. This rapid change may be classed as a 

discontinuity. 

Graph c) shows the second differential of the correlation surface 
in the x direction. It will be seen that the sharp correlation maximum 
16 that produced a rapid change in the first differential about point 

30 20 has produced a large value of the second differential at point 26. 

In contrast, the less sharp correlation maximum at point 14 with its 
more gradual change in the first differential at point 18 has produced 
a lower peak 28 in the second differential. This difference is used to 
identify "correlation maximum 16 as sharper since the second 

35 differential exceeds the second differential threshold value TH 6 . 

The circuits illustrated in Figures 6, 7. and 8 form part of the 
motion vector estimator 4 illustrated in Figure 1. As shown in Figure 



6, correlation surface data from the direct block matcher 3 of Figure 
1 is received at an input node 30 and stored in a correlation surface 
store 32. The correlation surface data values CS are then sequentially 
read out of the correlation surface store 32 under control of a read 
address generator 34. I? the read address generator 34 produces 
addresses raster scanning through the correlation surface along lines 
in the X direction, then this is the direction in which subsequent 
first and second derivatives are taken. If the read address generator 
34 is modified to scan in the Y direction, then the same circuitry will 
take first and second derivatives in the Y direction. 

The correlation surface values CS read from the correlation 
surface store 32 are fed to a subtracter 36 both directly at a 
subtracter input 38 and indirectly at a subtracter input 40 via a one 
value delay unit 42. The action of the subtracter 36 is to calculate 
a difference between adjacent correlation surface values CS. Such a 
difference is representative of the first derivative of the correlation 
surface along the scanning direction x. 

The output from the subtracter 36 that represents the first 
derivative is fed to a subtracter 44 both directly at a subtracter 
input 46 and indirectly at a subtracter input 48 via a one value delay 
unit 50. The action of the subtracter 44 is to calculate a difference 
between adjacent first derivative values. This difference is 
representative of the second derivative of the correlation surface 
values CS in the x direction. 

The output from the correlation surface store 32 is also fed to 
a minimum value comparator 52 and a lowest value store 54. If the 
lowest value comparator 52 determines that the current correlation 
surface value CS is lower than that currently stored in the lowest 
value store 5*1, then the lowest value store is write enabled with a 
signal W to replace its previous contents with that of the current 
correlation surface value CS. After the entire correlation surface has 
been scanned out of the correlation surface store 32, the lowest value 
store 54 will store the lowest correlation surface value within the 
correlation surface being analysed. 

The output from the subtracter 36 representing the first 
derivative value is fed directly to a positive value detector 56 and 
indirectly to a negative value detector 58 via a one value delay unit 
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60. The outputs from the positive value detector 56 and the negative 
value detector 58 are fed to a two input AND gate 62. In the case of 
a zero crossing going from a negative value to a positive value the 
outputs from the positive value detector 56 and the negative value 
detector 58 will be simultaneously asserted true and the output from 
the AND gate 62 will indicate a correlation maximum in the x 

direction. 

The output from the subtracter 44 that is representative of the 
second derivative is fed to a threshold value detector 64. If the 
second derivative exceeds a predetermined threshold value TH S then a 
sharpness flag is asserted true. 

The circuit of Figure 6 will be substantially duplicated to 
provide a circuit that simultaneously scans in the y direction and 
calculates and tests first and second derivative values in the y 
direction in the same way as those of Figure 6. The lowest value 
comparator 52 and the lowest value store 54 need not be duplicated. 

Figure 7 illustrates a circuit for processing the outputs of the 
circuit of Figure 6 as well as the corresponding (not illustrated) 
circuit performing simultaneous analysis in the y direction. The flags 
F xmin a* 1 ** Fynin indicating the presence of a correlation maximum together 
with the flags F xs and F ys indicating sharpness are fed to a four input 
AND gate 66. The output from the AND gate 66 F v indicating the 
detection of a sufficiently sharp correlation maximum is fed to one 
input of a two input AND gate 68 and one input of a three input AND 
gate 70. 

The correlation surface value CS currently being read from the 
correlation surface store 32 is supplied to the input of a best 
correlation maximum store 72 and via a multiplexer 74 to the input of 
a next best correlation maximum store 76. The output from the AND gate 
70 controls the multiplexer 74 such that if the output is false the 
input of the next best correlation magnitude store 76 is connected to 
the best correlation magnitude store 72 and if the output is true is 
connected directly to the correlation surface value CS. 

A first comparator 78 determines if the correlation surface value 
CS is less than the value currently stored in the best correlation 
magnitude store 72 (i.e. with a non-inverted surface a lower value 
indicates better correlation) . A second comparator 80 detects whether 



the correlation surface value CS is less than the value currently 
stored in the riext best correlation magnitude store 76. If the first 
comparator 78 gives a true result and the flag F v is true, then the AND 
gate 68 write enables the best correlation magnitude store 72 and the 
next best correlation magnitude 76 via an OR gate 82 ♦ If the 
correlation surface value CS is less than the value in the next best 
correlation magnitude store 76 but greater than the value in the best 
correlation magnitude store 12 and the flag F v is true f then the AND 
gate 70 write enables the next best correlation magnitude 76 via the OR 
gate 82 to directly receive the correlation surface value CS via the 
multiplexer 74. 

The two stores 72, 76 are initialised with default high values at 
the beginning of each* scan. 

The combined action of the elements of the circuit of Figure 7 is 
to detect and store the best and next best correlation maxima 
encountered during a complete scan* The best magnitude correlation 
store 72 can also include a section for storing the address value 
produced by the address generator 34 that is associated with the 
correlation value it is storing since this address value is indicative 
of the motion vector associated with that correlation maximum. 

The circuit of Figure 8 is operative at the end of the scan of a 
correlation surface to compare the best correlation maximum found with 
the lowest value stored in the lowest value store 54 using a comparator 
84. A subtracter 86 determines the difference between the correlation 
magnitude of the best correlation maximum encountered and the next best 
correlation maximum encountered and feeds this to a comparator 88 where 
a test is made as to whether this difference is above an alias 
threshold value TH A . If both of the outputs of the comparators 84, 88 
are true, then the AND gate 90 will reset a flag indicating that the 
correlation surface should not be used as a source of motion vectors. 

Figure 9 illustrates the operation of the circuit of Figure 7 in 
flow diagram form. Step 92 tests if all of the flags F^, F X8 , 
and F ys are true. If this condition is not met, then the process ends. 
If the condition is met, then the process passes to step 9** where the 
correlation surface value CS is compared with the current best 
correlation magnitude value CS b . If "CS < CS b ? lf ' is true, then the 
process passes to step 96 where the next best correlation magnitude 
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value is set to the current best correlation magnitude value and then 
the best correlation magnitude value is set to the correlation surface 
value. The process then ends. 

If the test at step 9*1 is false, then the process passes to step 
98 where a test is made as to whether the correlation surface value CS 
is less than the next best correlation magnitude value CS nb , If this 
condition is met, then the process passes to step 100 where the next 
best correlation magnitude value is set to the correlation surface 
value. If the test at step 98 is not met, then the process ends. 

Figure 10 illustrate the operation of the circuit of Figure 8 in 
flow diagram form. Step 102 tests if the best correlation magnitude 
value is equal to the lowest value found in the correlation surface. 
Unless this is the case the process ends. Step 10*f tests if the 
difference between the next best correlation magnitude value and the 
best correlation magnitude value exceeds a predetermined threshold TH A . 
Unless this condition is also met the process ends. If both the tests 
at steps 102 and 104 are met, then the process passes to step 106 where 
a flag that otherwise prevents any motion vectors being calculated from 
the correlation maxima of a correlation surface is reset. 

The overall operation of the circuits can be considered in the 
following way. The circuit of Figure 6 and the four input AND gate 66 
of Figure 7 serve to identify sufficiently sharp correlation maxima. 
The remainder of the circuit of Figure 7 is responsible for storing the 
best and next best correlation maxima encountered during the scanning 
of a complete correlation surface. The circuit of Figure 8 performs 
two further validity tests upon those sufficiently sharp correlation 
maxima that have been detected, i.e. that the lowest point in the 
correlation surface corresponds to the best correlation maximum 
detected and that the difference between the best correlation maximum 
detected and the next best correlation maximum detected exceeds a 
threshold value. 



11 

CLAIMS 



1. Apparatus for motion analysis of moving images, said apparatus 
comprising: 

means for calculating a correlation surface representing image 
correlation between a search block of image data within a current image 
and portions of a temporally adjacent image displaced by differing 
displacement vectors from said search block; 

means for identifying points in said correlation surface 
corresponding to correlation maxima; 

means for calculating a value of a second differential of said 
correlation surface in at least one direction at least said points of 
correlation maxima; s 

means for comparing said at least one value with a predetermined 
second differential threshold value in a threshold test; and 

means for rejecting any points of correlation maxima that fail 
said threshold test. 

2. Apparatus as claimed in claim 1, wherein said means for 
calculating calculates a value of a second differential of said 
correlation surface in two directions. 

3. Apparatus as claimed in claim 2, wherein said two directions are 
orthogonal . 

4. Apparatus as claimed in claim 3. wherein said two directions 
correspond to cartesian coordinate directions in said correlation 
surface. 

5. Apparatus as claimed in any one of the preceding claims, wherein 
said means for identifying calculates a value of a first differential 
of said correlation surface in two directions and identifies as 
correlation maxima points at which said values both pass from 
indicating a correlation increase to indication a correlation decrease. 

6. Apparatus as claimed in any one of the preceding claims, 
comprising: 
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means for detecting a correlation magnitude maximum point in said 
correlation surface at which correlation is at a maximum; and 

means for rejecting said correlation surface for use in motion 
analysis if said correlation magnitude maximum point does not 
correspond to a point of a correlation maximum that passed said 
threshold test. 

7. Apparatus as claimed in any one of the preceding claims, 
comprising: 

means for determining, in a correlation surface with at least two 
correlation maxima, a difference value of correlation magnitude between 
a point of a correlation maximum having greatest correlation magnitude 
and a point of a correlation maximum having next greatest correlation 
magnitude; and 

means for rejecting said correlation surface for use in motion 
analysis if said difference value is less than a predetermined 
difference threshold value. 

8. Apparatus for motion compensated image standards conversion 
including apparatus for motion analysis of moving images as claimed in 
any one of the preceding claims. 

9- A method of motion analysis of moving images, said method 
comprising the steps of: 

calculating a correlation surface representing image correlation 
between a search block of image data within a current image and 
portions of a temporally adjacent image displaced by differing 
displacement vectors from said search block; 

identifying points in said correlation surface corresponding to 
correlation maxima; 

calculating a value of a second differential of said correlation 
surface in at least one direction at at least said points of 
correlation maxima; 

comparing said at least one value with a predetermined second 
differential threshold value in a threshold test; and 

rejecting any points of correlation maxima that fail said 
threshold test. 
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10. Apparatus for motion analysis of moving images substantially as 
hereinbefore described with reference to the accompanying drawings. 

11. Apparatus for motion compensated image standards conversion 
substantially as hereinbefore described with reference to the 
accompanying drawings. 

12. A method of motion analysis of moving images substantially as 
hereinbefore described with reference to the accompanying drawings. 
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